In order to debug the periodic glitches on PR horizontal accelerometers - that introduce a rotation of the F0 along ty and generate interferometer unlocks - a series of tests have been performed on Sa_PR control system during the last weeks:

• The sensing element of the accelerometers were re-centered.
• In order to identify possible communication problems on RapidIO bus, 'keep sample' has been enabled on coil driver DSP board code.
• In order to identify possible issues due to the distribution box, channels used by the in-air cables were swapped: horizontal accelerometers previously connected to channels 1,2,3 were moved to channels 4,5,6 and vertical accelerometers from #4,5 have been moved to #1,2.
• In order to identify possible issues due to the dedicated DSP boards, horizontal accelerometers have been connected to the DSP board previously connected to the vertical ones and viceversa. This operation has required a re-design of the controllers.
• In order to identify possible issues due to cabling, mechanical action on in-air cables was performed. Even though this test introduced similar glitches in the signals, they were longer respect to the ones causing the unlocks. Moreover similar actions on the accelerometer cabling of other towers produced similar results in ther respective signals.

Unfortunately all these were inconclusive. Even though the third test did not produce any change, we decided in any case to replace the distribution box this morning. Primary, secondary and feedback coils have been measured as well.

Boot image version of BPC DSP boards has been updated to the same version currently used on Sc_NI crate (see entry #62523).

Due to the power outages on Friday, first at MC building - that was initially recovered on the same morning - and then in the central building (see entries #62359 and #62360) the entire suspension control system (SAT+PAY+INJ+DET) and the BPC had to be kept off over the weekend. The full recovery started on Monday morning. Three main issues were found: 

• A DSP board connected to MC PSDi (serial #43168) was not able to boot. Its EEPROM was re-programmed and the board was re-installed.
• SR smart PDU failed to restart and its firmware has to be re-flashed, in order to power up the DSP crates.
• BS DSP connected to vertical accelerometers (serial #41032) went in a frozen state quickly after boot: GPS time was wrong and all acquired signals were constant. It has to be replaced with a spare (serial #43154). For that reason the accelerometer controllers have to be re-calculated and the vertical inertial damping gains re-estimated.

Now all hardware, control loops, DAQ and GC communication of BPC, BS, PR, NE, WE, NI, WI, IB, MC, OB are working properly except for BS vertical inertial damping. 

Sa_SR crate that was partially damaged by the blackout has been replaced with a spare. Now DSP master board (serial #43147) is back in slot #1 and MCH previous configuration has been restored in order to properly propagate the timing signals to the other DSPs. However, probably due to minor mechanical incompatibilities in the MicroTCA connectors, p4 coil driver board (serial #42074), connected to F0, has to be moved from slot #5 to slot #6: DSP code has been changed accordingly.  All SR loops are now closed.

The failure of the electric system on last Wednesday has required a complete recovery of the control electronics of all suspensions. At the start of the intervention on Thursday morning, all crates were ON, expect for SR ones where the smart PDU was frozen in an unknown state that required a reconfiguration of the device. Three DSP boards were damaged and has to be replaced with spares: 

• Sa_SR master board (serial #43131), connected to bottom ring LVDTs, was unreachable on the network and has been replaced with DSP #43147;
• Sc_BS PSD board (serial #42110), connected to marionette transversal optical lever, was unreachable on the network and has been replaced with DSP #45270;
• Sa_PR board (serial #41026), connected to both F0 horizontal and vertical LVDTs, was malfunctioning generating corrupted signals, probably due to a damaged ADC or DAC. It has been replaced with spare with serial #41049. 

Moreover the SR top crate was partially damaged by the blackout: the slot #1 that hosts the master board, connected to both DAQ and timing box, was not working properly anymore. In particular MCH was not capable of powering any DSP inserted in that slot (in order to exclude problems on the MCH itself, bottom and top MCHs were temporarily exchanged). For this reason the spare LVDT board has to be installed on slot #7, action that required a reconfiguration of the MCH in order to properly propagate the timing signals to the other DSPs.

After the hardware interventions all crates of all towers has to be resynchronized with timing for proper DAQ transmission and all controls loops has to be verified and closed. The system was fully operational on Friday evening as reported by the on-shift operator (entry #60260).

As mentioned in entry #59501, this morning at around 8:30 UTC, during some tests on PDU new supervisor, the WE top crate was incidentally turned off. This unintended operation unfortunately caused a problem on the DSP board connected to the F0 vertical accelerometers (serial #43154). The board has been replaced by the one previously connected to MC vertical accelerometers (serial #43158) before the installation of the current board setup (see entry #59222). New controllers have been estimated and implemented. A measurement of the vertical inertial damping control gains is now required before closing the loop.

Yesterday in order to enable inertial damping control on both horizontal and vertical DOFs of the MC suspension, after the issue described in entry #59222, the accelerometer DSP boards have been replaced with an LVDT (serial #43129) and Coil Driver p4 (serial #42119) set. This intervention required the production and installation of a dedicated distribution box based on the design of the one installed at SR. Due to the DSP board configuration changes, for all the 5 accelerometers, the demodulation phases of the LVDTs and the transfer functions have to be re-measured. Moreover the LQG controllers have to be re-designed and implemented in the CD board. This latter task has been completed this morning. 

The following devices are currently available under ESQB1: